package FIT::Document::Parser::Text;

use FIT::Class;
use FIT::Util::SpecialChars;
use utf8; 

# TODO add some syntax checking and tests

sub html_to_text {
    my $html = pop;
#print "# [$html] BEFORE CONVERSION\n"; #-------------

    for (qw(
        remove_tags
        normalize_whitespace
        fold_whitespace
        normalize_newlines
        expand_entities
        expand_specials
        trim_whitespace
        unquote
    )) {
        $html = $self->$_($html);

#print "# [$html] AFTER STAGE $_\n"; # ----------------
    }
    return $html;
}

# TODO for templates- from perl string to something you can put in a cell
sub text_to_html {
    my $text = pop;
    return $text;
}

# remove everything except BR and /P
sub remove_tags {
    my $s = pop;
    $s =~ s{
        <(?!
            (?: \s* br \s* /? \s* >) |
            (?: \s* / \s* p \s* >)   
        ) [^>]*>  
    }{}gx;
    return $s;
}

sub normalize_whitespace {
    my $s = pop;
    for ($s) {
        s/\t/ /g;
        s/\n/ /g;
    }
    return $s;
}

sub fold_whitespace {
    my $s = pop;
    $s =~ s/\s+/ /g;
    return $s;
}

# note that to emulate firefox rendering, we trim whitespace
# after BR, and around /P, also 1 newline before /P
sub normalize_newlines {
    my $s = pop;
    for ($s) {
        s{\s*<\s*/\s*p\s*>\s*}{</p>}g ; # trim around /P 
        s{<\s*br[^>]*>\s*}{<br/>}g;     # trim after BR
        s{<br/>}{\n}g;                  # BR to newline
        s{\n?</p>(?:</p>)?}{\n}g;       # /P to \n, double \n folded
    }
    return $s;
}

sub expand_entities {
    my $s = pop;
    for ($s) {
        s/&quot;/"/g;
        s/&lt;/</g;
        s/&gt;/>/g;
        s/&amp;/&/g;
    }
    return $s;
}

sub expand_specials {
    my $s = pop;
    for ($s) {
        no warnings;
        s/$NBSP_REPLACEMENT/ /g;
        s/$ESCAPED_TAB/\t/g;
        s/$OCTET_ESCAPE([0-9a-fA-F]{2})/${\( chr hex $1 )}/g;
        s/$UNICODE_ESCAPE([0-9a-fA-F]{4})/${\( chr hex $1 )}/g;
    }
    return $s;
}

sub trim_whitespace {
    my $s = pop;
    for ($s) {
        s/\A\s+//;
        s/\s+\z//;
    }
    return $s;
}

sub unquote {
    my $s = pop;
    $s =~ s/$NO_TRIM_QUOTE//g;
    return $s;
}

sub prepare_for_read {
    my $s = pop;
    $s =~ s/&nbsp;/$NBSP_REPLACEMENT/g;
    return $s;
}

sub prepare_for_write {
    my $s = pop;
    $s =~ s/$NBSP_REPLACEMENT/&nbsp;/g;
    return $s;
}

=encoding utf8

=head1 NAME

FIT::Document::Parser::Text - extract accurate text from cell HTML


=head1 SYNOPSIS

    use aliased 'FIT::Document::Parser::Text';

    # before parsing HTML, prepare the string for read
    $html = Text->prepare_for_read($html);

    # now parse HTML using XML::Twig, get cell HTML using inner_xml and run
    # it though html_to_text to get the text representation

    print Text->html_to_text("a<br/>b"); # print a\nb

    # if needed, sprint the twig, but prepare it for before dumping to disk
    $html = Text->prepare_for_write($html);


=head1 DESCRIPTION

Use C<html_to_text()> to get a WYSIWYG text representation of HTML.


=head1 WHY

FIT testers need to get text from table cells. The text should represent
the look of the cell HTML. There should be an accurate way to specify any
list of octets in a table cell, despite the fact that WYSIWYG HTML editors
output almost random spaces and tags when user types in a cell.

Here is how the authors of FIT worded this requirement (from the FIT spec):

I<Fixtures may ask Fit to convert HTML into a string.  Fit strives to “render”
the HTML in the same way a browser would, so that fixtures see the same
result human readers do.>


=head1 CONVERSION DETAILS

This version of FIT uses an improvement over the basic FIT HTML to text
algorithm. We use UTF-8 encoded special unicode characters for out-of-band
data inside the cell. We also fold and trim differently, emulating how
firefox does it when rendering HTML.

=over

=item C<< spaces/tabs/newlines >>

Normalized to space, trimmed and folded. There are special trimming rules
around BR and /P tags: trim after BR, around /P.

=item C<< BR and closing P tag >>

Converted to newlines, not folded. Trimmed. Extra trimming of 1 newline before
/P. Like Firefox.

=item C<< ┓ >>

Converted to the tab character, not folded.

=item C<< &nbsp; >>

Converted not space, not folded. Trimmed.

=item C<< ‡ >>

Removed completely. Quote character. For when you don't want trimming.

=item C<< β >>

Follow with any octet, in 2 hex chars. E.g. β41 is capital letter A

=item C<< μ >>

Follow with any unicode code point, in 4 hex chars. E.g. μ05D0 is HEBREW
LETTER ALPHA. It will appear in the text as Perl native UTF-8.

=back

=head1 PREREQUISITES

There are several requirements of incoming HTML, which must be somewhat
normalized:

=over

=item 1.

Run HTML before Tidy'ing through C<prepare_for_read()>.

=item 2.

Run HTML before parsing through HTML::Tidy, with options tidy indent=auto,
bare, xhtml, utf8.

=item 3.

Parse with XML::Twig.

=item 4.

Empty P elements have been removed.

=back


=head1 TESTS

There are many examples in the unit test for this class. There is also a
file called C<< Browser_HTML_Rendering_Exploration.html >> in the directory
C<< FIT/spec >>, showing some non-intuitive cases of browser rendering.


=cut


